Systems and methods for a computer understanding multi modal data streams

ABSTRACT

Systems and methods for understanding (imputing meaning to) multi modal data streams may be used in intelligent surveillance and allow a) real-time integration of streaming data from video, audio, infrared and other sensors; b) processing of the results of such integration to obtain understanding of the situation as it unfolds; c) assessing the level of threat inherent in the situation; and d) generating of warning advisories delivered to appropriate recipients as necessary for mitigating the threat. The system generates understanding of the system by creating and manipulating models of the situation as it unfolds. The creation and manipulation involve “neuronal packets” formed in mutually constraining associative networks of four basic types. The process is thermodynamically driven, striving to produce a minimal number of maximally stable models. Obtaining such models is experienced as grasping, or understanding the input stream (objects, their relations and the flow of changes).

BACKGROUND

Over the last several decades, a variety of techniques have beendeveloped in Machine Intelligence and other disciplines dedicated toimage understanding, but the level of understanding accomplished by suchtechniques has been limited.

SUMMARY

The level of understanding accomplished by Machine Intelligence andother disciplines dedicated to image understanding has previously beenlimited to identifying and recognizing individual objects in the image.By contrast, human understanding does not stop at recognizing objectsbut extends to hypothesizing relationships between objects andreconstructing the overall situation, as manifested in the image orsequences of images. Moreover, situational understanding is facilitatedwhen visual information is combined with inputs in other sensorymodalities. Understanding involves composition of mental models thatenable predicting possible future changes in the situation as well asretrodicting earlier conditions from which the present situation couldhave developed. Systems and methods that emulate understandingmechanisms in the humans and allow reconstructing complex situationsfrom multi modal data streams will find applications in intelligentmonitoring, cyber warfare and other areas. Another class of applicationsinvolves dynamic structuring of information displays in human-machinesystems for the purpose of facilitating operator understanding ofcomplex dynamic situations and enabling robust decision making in realtime.

Human cognition has evolved to optimize performance in fluid and complexenvironments. Unlike in a photograph or other fixed images, objects andconditions in the environment are not frozen in time but undergocontinuous changes. As a result, understanding an image is notrestricted to recognizing objects but involves mentally complementingthe present fixed image with a series of other images reflectingconditions that are likely to precede and follow those in the image.That is, understanding is contingent on creating mental models capableof predicting future and retrodicting past conditions so that thepresent image can be seen as capturing some intermediate stage in anunfolding situation. When processing streaming data, such models arecreated and adjusted dynamically, based on the degree of mismatchbetween the predicted and actual conditions.

According to the present disclosure, understanding a situation involvesreducing combinations of objects and conditions into a two-partite orthree-partite model expressing, accordingly, two or three components anda relationship between them, such as “John loves Mary,” “USA suppliestroops to coalition forces”, “bottle contains milk,” “in this chessposition, whites are threatening blacks in the center,” “target Adeploys weapon B to attack asset C,” etc. Models can be nested, that is,any component in a model can be a composition of other models.Components are defined by spatial characteristics (e.g., size,location), temporal characteristics (e.g., the time of appearance) andfeatures (color, texture, amount, etc.). Relationships are defined bythe manner in which the behavior of one component (sequence of events,or pattern, involving changes in the component's features and spatialcharacteristics over time) constrains behavior of the other component.For example, the relationship “love” defines a pattern such thatco-location of the components in the past entails their likelyco-location over extended time intervals in the future, the relationship“attack” defines a pattern where an object with features and spatialcharacteristics of a weapon enters the vicinity of an asset, etc.

In situational models, one component remains an invariant part of therelationship persisting throughout the situation or several situationswhile the other components are subject to variation. For example, in“bottle contains milk” the component “bottle” can be consideredinvariant while the contents are allowed to vary. Varying the amount ofmilk can be considered continuing the on-going situation while replacingmilk with, say, water defines the onset of a new one. In a similar way,in “John loves Mary,” John seeks co-location with another person, aninstance of that relationship (a situation) continues for as long asthat person remains to be Mary and until either Mary is replaced or thebehavior ceases. Relationships can be inverted, for example, the inverseof “containment” is “to be contained.” In that inverse, “milk” can bethe invariant part while the container is allowed to vary (milk istransferred from bottle A to bottle B). In “being loved,” Mary can bethe invariant component engendering seeking behavior in varying parties.

A computer based system capable of constructing situational modelsreceives input from different sensors and utilizes several types ofartificial neurons for input processing, as follows: sensory neurons(pixels) respond to different elements (features) in the incomingstreams, temporal neurons respond to various temporal relations in theactivation of sensory neurons (e.g., activation of neuron A precededactivation of neuron B), spatial neurons respond to different locationsand relative positions of activation sources (e.g., neuron A wasactivated by sensor located in X), complex neurons respond to variousactivation patterns involving sensory, temporal and spatial neurons,hypercomplex neurons respond to various compositions of activationpatterns involving complex neurons, and semantic neurons respond tovarious patterns of activation involving hypercomplex neurons andassociate such patterns with labels in a finite set of labels signifyingmeaningful relationships (“love,” “threat”, etc). For example, semanticneuron “containment” can be defined by a pattern including two objectsappearing separately, followed by placing one object inside the volumeoccupied by the other object, followed by separating the objects (and/orother alternatives). Each such object can be defined by featureconstellations, spatio-temporal characteristics and behavior patternsspecified in hypercomplex and complex neurons. All neurons respondprobabilistically, that is, their activation by the correspondingpattern can be less than 100% certain. Sets of features activatingdifferent sensory neurons (neuron's sensitivity range) can partiallyoverlap. In the same way, patterns activating different complex andhypercomplex neurons can partially overlap.

Input processing for situational understanding involves two interrelatedstages: I) identifying (recognizing) objects and conditions and II)constructing a two- or three-partite model hypothesizing an overallrelationship between the objects and conditions and thus determining thesituation as the situation unfolds.

Stage I constructs an associative network where nodes are sensoryneurons and links between the nodes are weighted by the relativefrequency of neuron's co-activation (each time neurons N₁ and N₂ areboth activated within some time window, the weight of the linkconnecting N₁ and N₂ is incremented. Alternatively, absence ofco-activation causes the weights to decay). Next, the network ispartitioned into maximally internally cohesive (maximum summary weightof the internal links) and minimally externally coupled subnetworks(minimal summary weight of the external links connecting the subnetworkto other such subnetworks). For each subnetwork, boundary energy barrieris computed as a function of the ratio of the sum of the internal to thesum of the external weights in that subnetwork. Subnetworks surroundedby energy barriers at their boundaries are called “neuronal packets.”Packets behave as units: activation of any neuron in a packet producesactivation in the rest of the packet. Activation patterns causingresponses in complex neurons are paths in the network of sensorypackets. Similarly, activation patterns causing responses inhypercomplex neurons are paths in the network of complex neurons. Due tointerplay between activation and decay, link weights in all the networksare continuously updated entailing periodic re-partitioning in thenetworks and re-shaping of the packets. The barriers play a pivotal rolein such re-partitioning, as follows.

Removing a neuron from a packet requires energy expenditure (incursenergy cost) while absorbing a neuron into a packet returns energy intothe network with both expenditures and returns being determined by theheight of the barrier and the relative strength (the sum of linkweights) with which the neuron crossing the barrier is attached to otherneurons. When neurons are absorbed, some fixed percentage of releasedenergy is lost as a result of dissipation. Each partitioning cyclereceives fixed energy budget and seeks the best partitioning (minimizingthe total external link weight to the total internal link weight ratioacross the network) obtainable within that budget (the sum ofexpenditures and dissipative losses remains within the budget). As aresult, in each re-partitioning cycle packets surrounded by higherbarriers tend to undergo marginal modifications (shedding weaklyattached neurons) while those surrounded by lower barriers can dissolveand their constituent neurons can be absorbed into the surroundingpackets. In this way, energy barriers determine the relative packetstability.

Stable packets of sensory neurons define persistent featureconstellations characteristic of particular objects thus allowing“object recognition”. Inclusion of spatial neurons in a packet candefine a particular location, or spatial tag (object A in location X),inclusion of a temporal neuron can define a particular time interval, ortemporal tag (object A at time T₁). Complex neurons can be defined thatrespond to packets with spatial and temporal tags and assert theobject's identity (object located at X₁ at time T₁ and at X₂ at time T₂is the same object, object A).

Complex and hypercomplex neurons organize into associative networkspartitioned into packets, in the same way as sensory neurons organizeinto a network partitioned into packets. Stable packets of complexand/or hypercomplex neurons define repetitively co-occurringcombinations of patterns.

In the networks of sensory, complex and hypercomplex neurons, linkweights are modified only when neurons connected by the link getco-activated, or as a result of decay in the absence of co-activation.Combining neurons into packets, removing neurons from and/or absorbingneurons into packets entails no weight modification.

Activation patterns associated with complex, hypercomplex and semanticneurons are initially defined by the system user. However, when stablepackets of sensory neurons emerge, new complex neurons can be generatedresponding to various patterns of packet activation different from thoseenvisioned by the user. In time, co-activated complex neurons getassociated into link-weighted networks that are subsequently partitionedinto packets, in the same way as the networks of sensory neurons arepartitioned into packets. Similarly, new hypercomplex neurons can begenerated responding to activation patterns involving packets of complexneurons.

Stage II uses packets produced at stage I to construct situation models.Models are formed of groups of packets of hypercomplex neurons, modelconstruction involves shifting neurons between packets, adjustingresponses of individual neurons (inhibiting responses to some complexneurons and amplifying responses to other complex neurons within thehypercomplex neuron's sensitivity range), and combining packets intogroups, until a two- or three-partite configuration is constructed(situational model) that is a) comprised of two or three groups ofhypercomplex neurons and b) causes activation of at least one semanticneuron. Groups in the model are not restricted to paths in a network ofhypercomplex packets but can include any combination of such packets(for example, a group formed on path A-B-C can include packets A and Bbut not C). In this way, a model can hypothesize a relationship betweenobjects and/or events that are separated by arbitrary time intervals andany number of intervening objects and/or events. Groups are not packets,that is, no energy barriers are formed at their boundaries. Accordingly,adding to or removing packets from groups incurs no energy costs.

Models are validated by comparing predicted events to those actuallytaking place in the course of time. For example, model “target A deploysweapon B to attack asset C” predicts movement of B in the direction ofC. Disconfirming the model (no such movement is observed) entailsre-construction which can involve deselecting packets, selecting newones and their subsequent adjustment. Repetitive prediction failurescause iterative re-grouping and adjustment of packets in hypercomplex,complex and sensory networks. Similar to processes in stage I, eachmodeling cycle receives a limited energy budget for packet adjustmentsand seeks to find the most efficient adjustments. Efficient adjustmentsreduce the amount and expense of subsequent adjustments, the efficiencyis maximized when models can be formed by grouping packets, with minimalor no packet adjustments.

In probabilistic networks partitioned into packets, network entropy canbe computed as a function of the ratio of the total weight of interpacket links to the total weight of all the packets in the network.Seeking partitions that minimize the ratio equates to entropy reduction,or negentropy production. The overall method for processing multi modaldata streams can be defined as iterative entropy reduction in a systemof networks, with each iteration cycle investing limited energy budgetsinto entropy reducing operations in the network.

Accordingly, described below are non-limiting example embodiments ofvarious aspects of systems and methods for processing multi modal datastreams.

A computer implemented method in a self-adaptive multi modal data streamprocessing system having at least one computer processor and at leastone spatiotemporal associative memory coupled to the at least onecomputer processor may be summarized as including constructing, by aconstruction module under control of a control module of the multi modaldata stream processing system, a model of a situation built upon anunderlying associative neural network stored in the associative memoryto establish situational understanding of the situation, wherein themodel is comprised of invariant and variable entities and relationshipsbetween the entities, wherein each entity is able to be nested by thecontrol module by being comprised of lower level models and wherein thelower level models are formed of packets and are groups of packets;manipulating the lower level models by the control module of the multimodal data stream processing system, leaving the underlying associativeneural network intact by not changing synaptic weights in the underlyingassociative neural network in the manipulation of the lower levelmodels; and reducing, by the multi modal data stream processing system,energy consumption and energy dissipation in the constructing and themanipulating of the models by the control module seeking progressivelymore general and adequate models persisting through various situationsand wherein the reducing energy consumption and dissipation translatesinto negentropy production.

Manipulating the lower level models may include grouping the packets;adjusting the packets, including adding or removing artificial neuronsof the underlying associative neural network in the associative memory;and varying packet responses of the packets, including rotating a vectorof neuronal responses inside the packets.

A computer-implemented method in a self-adaptive multi modal data streamprocessing system having at least one computer processor and at leastone spatiotemporal associative memory coupled to the at least onecomputer processor may be summarized as including constructing, by acontrol module of the multi modal data stream processing system, atleast one three-partite situation model of a situation represented by aplurality of different data streams, by making associations ofartificial neurons of a plurality of artificial neurons of variousdifferent types in an artificial neural network in the spatiotemporalassociative memory, wherein the constructing of the at least onethree-partite situation model includes: developing, by a control moduleof the multi modal data stream processing system, link-weightedassociative artificial neural networks in the spatiotemporal associativememory, wherein the developing the link-weighted associative artificialneural networks includes: corresponding, by the multi modal data streamprocessing system, individual nodes of the link-weighted associativeartificial neural networks to respective artificial neurons of theplurality of artificial neurons that respond to different data elementsof data representing the plurality of different data streams; andestablishing link weights of the link-weighted associative artificialneural networks which represent a frequency of co-occurrence of thedifferent data elements of the data representing the plurality ofdifferent data streams; dynamically partitioning as the situationunfolds over time, by the control module of the multi modal data streamprocessing system, the link-weighted associative artificial neuralnetworks into internally cohesive subnetworks and externally weaklycoupled subnetworks, wherein the subnetworks are packets, eachcorresponding to at least a respective one of various differentcombinations of the data elements; performing dynamic mapping, by thecontrol module of the multi modal data stream processing system, betweenthe packets as the situation unfolds over time to adjust the at leastone three-partite situation model to improve the at least onethree-partite situation model for use in understanding of the situation,wherein the partitioning and dynamic mapping leave the artificial neuralnetwork intact by not changing synaptic weights in the artificial neuralnetwork in the partitioning and the dynamic mapping; based on the atleast one three-partite situation model, generating, by the multi modaldata stream processing system, situational understanding of thesituation; and reducing, by the multi modal data stream processingsystem, energy consumption and dissipation in the constructing,partitioning and dynamically mapping by the control module seekingprogressively more general and adequate models persisting throughvarious situations and wherein the reducing energy consumption anddissipation translates into negentropy production.

Performing dynamic mapping between the packets as the situation unfoldsover time may include associating, by the multi modal data streamprocessing system, the packets into distinct packet groupingsrepresenting situational entities of the situation; associating, by themulti modal data stream processing system, the packet groupings intoinvariant situational components of the situation; associating, by themulti modal data stream processing system, packet groupings intovariable situational components of the situation; assigning, by themulti modal data stream processing system, a relationship type to eachrespective situational component of the invariant situational componentsand to each respective situational component of the variable situationalcomponents based on contents of and behavior of the respectivesituational component to construct the at least one three-partitesituation model; manipulating the at least one three-partite situationmodel; comparing results of the manipulation with relevant data of theintegrated data stream to facilitate determining validity of the atleast one three-partite situation model; and adjusting the at least onethree-partite situation model based on the comparing.

The method may further include receiving, by the multi modal data streamprocessing system, the plurality of different data streams, each of thedifferent data streams originating from different sensors; and inresponse to the receiving the plurality of different data streams,automatically integrating, by the multi modal data stream processingsystem, the plurality of data streams in real time to produce anintegrated data stream; and based on the generated situationalunderstanding, the multi modal data stream processing system generatingwarning advisories for delivery to relevant selected recipients tomitigate a threat posed by the situation. Constructing the at least onethree-partite situation model of a situation represented by theplurality of different data streams may occur in response to theintegrating the plurality of data streams in real time to produce anintegrated data stream. Performing the dynamic mapping between thepackets as the situation unfolds over time may include automaticallyassessing, by the multi modal data stream processing system, a level ofa threat existing in the situation as the situation unfolds over time.

Assessing the level of a threat existing in the situation as thesituation unfolds over time may include predicting, by the multi modaldata stream processing system, changes in the situation using the atleast one three-partite situation model; and determining, by the multimodal data stream processing system, adequate responses. At least someof the different data elements may represent pixels. The frequency ofthe co-occurrence of the different data elements of the datarepresenting the plurality of different data streams may be correlatedto an associative strength between the artificial neurons of theplurality of artificial neurons of various different types. Theplurality of artificial neurons of various different types may include acombination of: sensory neurons, temporal neurons, feature neurons,spatial neurons, hyper complex neurons, semantic neurons, and controlneurons. The link weights of the link-weighted associative artificialneural networks may reflect co-occurrence of state changes in theplurality of artificial neurons of various different types. Performingdynamic mapping between the packets as the situation unfolds over timemay include computing, by the multi modal data stream processing system,response strength of each respective artificial neuron of at least someof the artificial neurons of the plurality of artificial neurons as afunction of activation probabilities of artificial neurons connected tothe respective artificial neuron. Performing the dynamic mapping betweenthe packets as the situation unfolds over time may include inserting orremoving links or nodes in the link-weighted associative artificialneural networks. Performing the dynamic mapping between the packets asthe situation unfolds over time may include adjusting the link weightsof the link-weighted associative artificial neural networks. Performingthe dynamic mapping between the packets as the situation unfolds overtime may include expanding or shrinking the packets. Performing thedynamic mapping between the packets as the situation unfolds over timemay cause improvement in predictive or retrodictive performance of thethree-partite situation model for use in threat level assessment of thesituation.

A multi modal data stream processing system may be summarized asincluding at least one computer processor; at least one non-transitoryspatiotemporal associative memory coupled to the at least one computerprocessor; and at least one non-transitory memory communicativelycoupled to the computer processor having computer-executableinstructions stored thereon that, when executed by the computerprocessor, cause the computer processor to: dynamically partition, as asituation represented by a plurality of different data streams unfoldsover time, link-weighted associative artificial neural networks intointernally cohesive subnetworks and externally weakly coupledsubnetworks, wherein the subnetworks are packets, each corresponding toat least a respective one of various different combinations of dataelements of data representing the plurality of different data streams;perform dynamic mapping between the packets as the situation unfoldsover time to adjust at least one three-partite situation model toimprove the at least one three-partite situation model for use inunderstanding of the situation by the system, wherein the partitioningand dynamic mapping leave the artificial neural network intact by notchanging synaptic weights in the artificial neural network in thepartitioning and the dynamic mapping; generate situational understandingof the situation based on the at least one three-partite situationmodel; and reduce energy consumption and dissipation in the partitioningand the dynamically mapping by seeking progressively more general andadequate models persisting through various situations wherein thereducing energy consumption and dissipation translates into negentropyproduction.

The computer-executable instructions that may cause the computerprocessor to automatically asses—the level of threat existing in thesituation as the situation unfolds over time based on the at least onethree-partite situation model, may include computer-executableinstructions that, when executed, may cause the computer processor to,based on the generated situational understanding, automatically asses—alevel of a threat existing in the situation as the situation unfoldsover time based on the; based on the assessed level of threat, generatewarning advisories for delivery to relevant selected recipients tomitigate a threat posed by the situation; facilitate timely automaticdetection and evaluation of malware in a computer network, whereinvarious features of the computer network are represented by thelink-weighted associative artificial neural networks; and dynamicallyimprove deployment of countermeasures against the malware over time. Thecomputer-executable instructions that may cause the computer processorto facilitate timely automatic detection and evaluation of malware, mayinclude computer-executable instructions that, when executed, may causethe computer processor to identify correlations between elements of themalware, geospatial locations within the computer network and temporalorganization of attack patterns of the malware revealed by the dynamicpartition of the link-weighted associative artificial neural networksinto internally cohesive subnetworks and externally weakly coupledsubnetworks.

A non-transitory computer readable storage medium, havingcomputer-executable instructions stored thereon that when executed by acomputer processor may cause the computer processor to construct, by acontrol module of the multi modal data stream processing system, atleast one three-partite situation model of a situation represented by aplurality of different data streams, by making associations ofartificial neurons of a plurality of artificial neurons of variousdifferent types in an artificial neural network in a spatiotemporalassociative memory; dynamically partition, as the situation unfolds overtime, by a control module of the multi modal data stream processingsystem, link-weighted associative artificial neural networks of theartificial neural network into packets, each corresponding to at least arespective one of various different combinations of data elements ofdata representing the plurality of different data streams; performdynamic mapping, by a control module of the multi modal data streamprocessing system, between the packets as the situation unfolds overtime to adjust the at least one three-partite situation model to improvethe at least one three-partite situation model for use in understandingof the situation, wherein the constructing, dynamically partitioning anddynamic mapping leave the artificial neural network intact by notchanging synaptic weights in the artificial neural network in theconstructing, partitioning and the dynamic mapping; generate situationalunderstanding of the situation as the situation unfolds over time basedon the at least one three-partite situation model; and reduce energyconsumption and dissipation in the constructing, dynamicallypartitioning and dynamically mapping by seeking progressively moregeneral and adequate models persisting through various situationswherein the reducing energy consumption and dissipation translates intonegentropy production. The computer computer-executable instructions,when executed by the computer processor, may further cause the computerprocessor to automatically asses—a level of a threat existing in thesituation as the situation unfolds over time based on the situationalunderstanding facilitate automated intelligent surveillance of thesituation. Automatically assessing a level of a threat may includeidentification of objects or conditions in the situation that constitutethe threat when acting in coordination. Automatically assessing a levelof a threat existing in the situation may facilitate a robotic vehicleor other robotic system automatically responding to an environment ofthe robotic vehicle or other robotic system to perform one or more of:pursuing specified objectives and responding to obstacles.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

In the drawings, identical reference numbers identify similar elementsor acts. The sizes and relative positions of elements in the drawingsare not necessarily drawn to scale. For example, the shapes of variouselements and angles are not drawn to scale, and some of these elementsare arbitrarily enlarged and positioned to improve drawing legibility.Further, the particular shapes of the elements as drawn are not intendedto convey any information regarding the actual shape of the particularelements, and have been solely selected for ease of recognition in thedrawings.

FIG. 1 is a block diagram of a self-adaptive multi modal data streamprocessing system, according to one illustrated embodiment.

FIG. 2 is a functional block diagram of a computing device suitable forimplementing the self-adaptive multi modal data stream processing systemof FIG. 1, according to one illustrated embodiment.

FIG. 3 is a flow diagram illustrating a process of the multi modal datastream processing system, according to one illustrated embodiment.

FIG. 4 is a flow diagram illustrating a process of the multi modal datastream processing system including details regarding how associativeartificial neural networks of the system are developed, according to oneillustrated embodiment.

FIG. 5 is a flow diagram illustrating a process of the multi modal datastream processing system including details regarding dynamicallypartitioning link-weighted associative artificial neural networks of thesystem, according to one illustrated embodiment.

FIG. 6 is a flow diagram illustrating a process of the multi modal datastream processing system including details regarding improving athree-partite situation model of the system for use in understanding asituation represented by the model, according to one illustratedembodiment.

DETAILED DESCRIPTION

In the following description, certain specific details are set forth inorder to provide a thorough understanding of various disclosedembodiments. However, one skilled in the relevant art will recognizethat embodiments may be practiced without one or more of these specificdetails, or with other methods, components, materials, etc. In otherinstances, well-known structures associated with computing systemsincluding client and server computing systems, as well as networks,including various types of telecommunications networks, have not beenshown or described in detail to avoid unnecessarily obscuringdescriptions of the embodiments.

Unless the context requires otherwise, throughout the specification andclaims which follow, the word “comprise” and variations thereof, such as“comprises” and “comprising,” are to be construed in an open, inclusivesense, that is, as “including, but not limited to.”

Reference throughout this specification to “one embodiment” or “anembodiment” means that a particular feature, structure or characteristicdescribed in connection with the embodiment is included in at least oneembodiment. Thus, the appearances of the phrases “in one embodiment” or“in an embodiment” in various places throughout this specification arenot necessarily all referring to the same embodiment. Furthermore, theparticular features, structures, or characteristics may be combined inany suitable manner in one or more embodiments.

As used in this specification and the appended claims, the singularforms “a,” “an,” and “the” include plural referents unless the contentclearly dictates otherwise. It should also be noted that the term “or”is generally employed in its sense including “and/or” unless the contentclearly dictates otherwise.

The headings and Abstract of the Disclosure provided herein are forconvenience only and do not interpret the scope or meaning of theembodiments.

FIG. 1 is a block diagram of a self-adaptive multi modal data streamprocessing system 102, according to one illustrated embodiment.

The self-adaptive multi modal data stream processing system 102 isconfigured to understand (impute meaning to) multi modal data streamssuch as those originating from data stream source(s) 104. In oneembodiment, the self-adaptive multi modal data stream processing system102 is used in intelligent surveillance and allows a) real-timeintegration of streaming data from video, audio, infrared and othersensors, b) processing the results of such integration to obtainunderstanding of the situation as it unfolds, c) assessing the level ofthreat inherent in the situation, and d) generating warning advisoriesdelivered to appropriate recipients as necessary for mitigating thethreat. Such situational understanding for the purposes of threatassessment can allow identification of objects and conditions thatappear innocuous when considered independently from each other but mightconstitute a threat when acting in coordination.

In another embodiment, the self-adaptive multi modal data streamprocessing system 102 is used to obtain a high degree of autonomy inrobotic vehicles or other robotic systems, endowing such systems withthe capability to pursue user-specified objectives and efficientlyrespond to obstacles unknown to the user as well as other unforeseeableeventualities.

A computer-implemented process in the self-adaptive multi modal datastream processing system 102 for understanding streaming data involvesconstructing three-partite situation models capturing 1) invariant and2) variable components of the situation and establishing 3) a meaningfulrelationship between the components. Operations on such models allowpredicting changes in the situation and composing adequate responses.For example, a situation model can include high value target(s) underprotection as the invariable component A and the means of attack as thevariable component B. Meaning is defined by the relationship “ATHREATENS B” which can be established when conditions exist allowing themeans to reach the targets, with the variety of such conditions andattack scenarios revealed by the simulation.

Situational models are constructed as associations of “neurons” ofvarious types, employing a process that emulates mechanisms in the humanbrain underlying understanding (mental modeling). In one embodiment,such construction involves 1) development of link-weighted associativenetworks such that nodes correspond to neurons responding to differentdata elements (e.g., pixels) and link weights represent the frequency ofsuch elements co-occurrence in the data stream (associative strength),2) dynamic partitioning of such networks into internally cohesive andexternally weakly coupled subnetworks (packets) corresponding to variouscombinations of data elements and 3) conducting dynamic mapping betweenthe packet networks which results in a) associating packets intodistinct groupings representing situational entities, b) associatingpacket groupings into stable (invariant) and changing (variable)situational components, c) assigning a relationship type to thecomponents based on their contents and behavior (creating a model), d)manipulating the model and comparing results with the data to determinethe model's validity (simulation) and e) adjusting the model based onthe simulation's outcome. Packet manipulation leaves the underlyingassociative network intact. That is, an input stream modifies synapticweights while packet manipulation does not. All animals are capable ofpattern detection, to a varying degree (this is called “conditioning”)while mental modeling is unique to the humans. Modeling by the system102 enables construction of higher order regularities, or invariants(laws of nature) while associations alone including, multiple othermethods in the class of “neural networks” can only detect transient“regularities.” Modeling (packet manipulation) by the system 102 buildson the foundation of associative networks but is by no means reducibleto it.

For example, relationship “THREAT” can be assigned to components A and Bwhen component A is determined to be moving towards component B, themodel predicts continuing movement until A enters the vicinity of B andgets adjusted when the approach is terminated followed by component Amoving away from component B. The architecture of the system 102 asshown in FIG. 1 includes a plurality of modules including an AssociativeMemory Module 106 comprised of “artificial neurons” and responsible fororganizing the said neurons into networks, a Construction Module 108responsible for partitioning such networks into packets and associatingthe said packets into groups (situational entities) and groups of groups(situational components) and Control Module 104 responsible forassigning relationship to the situational components and manipulatingthe said components in a manner determined by the assigned relationshipand used to validate the model by predicting the future and retrodictingthe past data elements in the data stream and assessing the degree offit between the model-generated and the actual data.

In one embodiment, the self-adaptive multi modal data stream processingsystem 102 imputes meaning to data streams (creates situationalunderstanding by the self-adaptive multi modal data stream processingsystem 102), by constructing models of entities and conditions likely toexist at the source of the streaming data (situation models). Theself-adaptive multi modal data stream processing system 102 emulatesneuronal mechanisms in the human brain responsible for creating andexercising models of the environment (mental models). The Control Module104 establishes a system of “artificial neurons” and associates dataelements and various combinations of data elements with such neuronsaccording to actions which include but are not limited to, partitioningstream history into equal length time intervals (episodes) andestablishing neurons of various types. Different types of neuronsrespond to: A) sensory features in the input (color (optical wavelength), sound (acoustic wave length), etc.), B) spatial features(location of the source of sensory inputs), C) temporal features(precedence, order of the sensory inputs) and D) combinatorial features,i.e., various combinations of the A, B and C feature types.

A, B, C, and D form separate but interacting self-partitioning networks.Interaction involves mutually constraining mappings between packets inthe networks, resulting in (experienced as) construction (detection) of“objects” in the environment (input stream). “Object” is a cohesivegroup of sensory features (sensory packet) that keeps co-occurring(temporal packet) in contiguous locations (spatial packet).

Combinatorial, or semantic neurons capture relations between “objects”(packet groupings). There are only four underlying relations betweenpacket groupings: group X is separate from group Y, X partially overlapswith Y, X is identical (completely overlaps) with Y, X is a part of Y.There are only two basic forms of relations between objects: “object Xrelates to object Y” and “object X relates to object Y via Z.” Thelatter relation corresponds to extracting the common component Z(invariant) from overlapping X and Y (variables). There are all kinds ofsemantic relations between objects (John loves Mary, box rests on thetable, rains cause floods, . . . ) are expressed as (boil down to)combinations of A, B, C groupings in one of the two basic forms (e.g.,“X might be the cause of Y if Y is always preceded by X and X is alwaysfollowed by Y, etc.). Combinations can be expressed as chains(sequences) or matrices (sequences of sequences, nested sequences) ofgroupings.

For example, the types of neurons include: a) sensory neurons respondprobabilistically to various data elements (e.g., pixels and pixelstates), b) temporal neurons which respond probabilistically to temporalrelations between the data elements (e.g., state change of pixel Aprecedes, coincides or follows state change of pixel B), c) featureneurons which respond probabilistically selectively to state changes ofsensory neurons and associate such changes with various situationalprimitives in (features of) the data source (e.g., state change insensory neuron A indicates the presence of color “red” in the source,state change in sensory neuron B indicates the presence of an edge-likeshape in the source), d) spatial neurons which respond probabilisticallyto state changes in the feature and temporal neurons and associate suchchanges with spatial relations between situational primitives at thesource (e.g., object A is adjacent to object B, object C is located ontop of object D, etc.), e) complex neurons which respondprobabilistically to various combinations of state changes in thefeature, temporal and spatial neurons and associate such changes withpersistent combinations (entities) and spatio-temporal relations betweensuch entities at the source, f) hyper complex neurons which respondprobabilistically to various combinations of state changes in complexneurons and associate such changes with persistent entity groupings andcondition combinations at the source, g) semantic neurons which respondprobabilistically to various combinations of state changes in complexand hyper complex neurons and associate such changes with meaningfulrelations (semantic links) between entity groupings and conditioncombinations definitive of the overall situation (e.g., group Athreatens group B, group A supports group B, etc.), g) control neuronswhich respond to inputs arriving from the Control Module and act toinhibit or activate other neurons, and h) other types of neurons.

The construction module 108, under control of the control module 104,constructs link weighted networks comprised of neurons of various types(temporal network, feature network, spatial network) such that neuronsform nodes in the network, links reflect co-occurrence of state changesin the neurons (e.g., from active to dormant) and link weights aredetermined by the relative frequency of such co-occurrences. The controlmodule 104 dynamically partitions such networks stored in theassociative memory module 106 into maximally internally cohesive andexternally minimally coupled subnets (packets). The control module 104conducts dynamic mapping between the networks in order to obtain packetgroupings across the networks and associates such packet groupings withdistinct situational entities with unique identity manifested in thehistory of the data stream (e.g., entity A in episode X and entity B inepisode Y are one and the same entity). For the purposes of suchmapping, response strength of neuron X is computed as a function ofactivation probabilities of the neurons connected to (mapped to) neuronX.

The construction module 108, under control of the control module 104,constructs (possibly, nested) components of situation models, bygrouping situational entities into persistent (invariant) and changing(variable) entity groupings (groups of groups of packets). The controlmodule 104 imputes meaning to the situation by connecting components toone of the semantic neurons and thus establishing a semantic linkbetween the constituents of the situation and applies globalorganizational criteria, such as relative complexity, negentropyincrement, or other criteria in order to increase performance of theprocess and the result of model construction.

The control module also assesses the model's performance by varyingmodel parameters and assessing the degree of fit between the model'soutput and the present data, as well as the historic data in the datastream. Then the control module 104 adjusts the model by insertingand/or removing links and/or nodes in the networks, adjusting linkweights, expanding and/or shrinking packets, in order to improve themodel's predictive and retrodictive performance and thus achieve ahigher degree of situation understanding.

The dynamic mapping involves connecting each such packet grouping to aparticular complex neuron or a particular group of complex neuronsresponding strongly to combinations of state changes in the neuronsinside the constituent packets in the packet groupings. Constructing thecomponents of the situation models involves connecting groups ofentities to a particular hyper complex neuron or particular groups ofhyper complex neurons responding strongly to combinations of statechanges in the entities inside the constituent entity groupings. Thesystem imputing meaning to the situation involves selecting semanticneurons responding strongly to the combination of temporal and spatialrelations between and feature compositions inside those components.

The Associative Memory Module 106 is comprised of different types ofneurons and is responsible for a) organizing such neurons intolink-weighted networks, and b) partitioning such networks into packets.The Construction Module 108 is responsible for a) conducting dynamicmapping between the packet networks, b) constructing (possibly, nested)situation models, and c) exercising the models, by varying theirparameters, to obtain predictions and retrodictions of data in the datastream, The Control Module 104 is responsible for a) applying globalcriteria to optimize construction of situation models, b) assessingmodel's performance, and c) applying adjustments to the model to achieveperformance improvements. The Control Module 104 may also output data110 regarding the system's understanding and assessment of the situationas it unfolds over time based on the situation models generated by thesystem 102.

According to one embodiment, dynamic mapping between packet networks inthe Construction Module 108 involves internetwork, intranetwork, globaland semantic operations, as follows:

A) Internetwork operations determine pair wise relations between packetsand/or packet groupings within a network:

-   -   1. Packet A is separate from packet B; or    -   2. Packet A overlaps packet B.

B) Intranetwork operations determine pair wise relations between packetsand/or packet groupings in different networks:

-   -   1) Packet A in network X connects to packet B in network Y; or    -   2) Packet A in network X does not connect to packet B network Y.

The inter- and intranetwork operations establish distinct and uniqueentities and associate such entities with particular locations, episodesand characteristics (features) manifested in the course of such episodes(e.g., entity A appeared in episode N at location X and manifestedfeature combination f1, f2, . . . , fn).

C) Global operations determine global pair wise relations betweenentities, including:

-   -   1. Entity A is identical to entity B.    -   2. Entity A is a part of entity B.    -   3. Entity A conditions entity B (causes, enables, triggers,    -   etc.).    -   4. Entity A can co-occur (is compatible) with entity B.

D) Semantic operations involve grouping entities into situationalcomponents and establishing semantic relations between the components.

Adjustment operations in the Control Module engage control neurons andinvolve inhibiting/activating individual neurons, changing responsecharacteristics (probabilities) of individual neurons, redistributingneurons between packets, and re-distributing packets between entities.

Both self-partitioning into packets and packet manipulations in thesystem 102 are configured to strive for thermodynamic optimization, thatis, the entire system 102 strives to minimize internal energyconsumption. Manipulating packets is work—it requires energy andminimizing energy consumption enforces constructing “durable,” stablemodels (that is, capturing persistent regularities, invariants in theenvironment). Human understanding is ultimately an expression of thebasic thermodynamic laws albeit in a unique fashion. Thus, constructionof the model at each step strives to minimize entropy in the networksystem (maximize negentropy increments) leading to progressivelyreducing the amount of adjustments while improving the degree of fitbetween the model's output and the data. The construction of the modelby the construction module 108 reduces the number of unique entities andnarrows the range of their behavior variation throughout differentepisodes and situations in the unfolding history of the situation.

In accordance with an aspect of the disclosure, the systems and/orcomponents shown in FIG. 1 may contain discrete functional programmodules that might make use of an application programming interface(API), or other object, software, firmware and/or hardware, to requestor provide services of one or more of the other systems or componentswithin the self-adaptive multi modal data stream processing system 102and/or running on the same machine. For example, communication can beprovided over a communications medium between any one of the systems orcomponents shown in FIG. 1.

FIG. 2 shows an example computing device 120 suitable for implementingthe self-adaptive multi modal data stream processing system 102 andexecuting any one or more of the modules of FIG. 1, according to oneillustrated embodiment. For example, the computing device 120 is acomputing device 120 executing operations and performing functions ofthe control module 104, associative memory module 106 and/or theconstruction module 108.

In one embodiment, the computing device of FIG. 2 is a wirelesscomputing device (e.g., a wireless tablet device). In other exampleembodiments, the computing device 120 may take the form of one or moreof the following: a desktop computer, personal digital assistant (PDA),handheld personal computer, appropriately configured cellular telephoneor smart phone, other handheld communications device, personal computer(PC), distributed computing devices, a server computer, etc. Thecomputing device 120 may also include a transmitter and receiver, ortransceiver 216 under the control of the processor 218, and be coupledto one or more external or internal antennas (not shown) to wirelesslytransmit and/or receive information, such as to receive multi modal datastreams and output information regarding the situation and the system'sunderstanding of the situation. The multi modal data streams may also beinput via direct inputs 210 from various sensors coupled to thecomputing device via various wired and/or wireless connections to video,audio, infrared, light, radio frequency (RF) and other sensors. Suitablecomputing device 120 may, for example, also include wireless local areanetwork (WLAN) capability per IEEE specification 802.11b and/or otherwired or wireless communications capabilities suitable for receivingmulti modal data streams.

The computing device 120 may include a power button 204 and a userinterface, such as one or more buttons 226 and/or display 202, such as aliquid crystal display (LCD), which may or may not be touch-sensitive,to serve as both a user input device and graphical output device.

The computing device 120 also includes one or more processors 218 and anassociative memory 220 for implementing the associative memory module106 as described herein, and optionally one or more other additionalmemory devices such as random access memory (RAM) 222, Read Only Memory(ROM) 212, flash memory and/or electronically erasable programmable readonly memory (EEPROM), or other storage devices 214 (including one ormore hard drives and/or optical disk drives) coupled by one or moresystem buses 224 to store data and instructions for execution by theprocessor 218 to implement the self-adaptive multi modal data streamprocessing system 102 described herein. For example, the instructionsmay be those of the multi modal data stream processing system 102 orindividual components thereof that, when executed, perform theprocesses, routines and functions described herein. The computing device120 may also include an interface driver 208 to couple user entries atthe one or more buttons 224 or touch screen entries to the processor218, and/or a display driver 206 to couple display data from theprocessor 218 to the display 202, such as to display output dataregarding understanding by the system 102 of a situation as it unfoldsover time.

For example, in one embodiment, the self-adaptive multi modal datastream processing system 102 can be implemented in the cyber domain, tofacilitate timely detection and evaluation of malware and to dynamicallyoptimize the deployment of countermeasures. Network attacks that areparticularly difficult to mitigate can be comprised of multiple malwareelements entering the network at different times and from differententry points. In such attack scenarios, each individual malware elementcan be harmless while their assembly can have a highly destructiveimpact on the network. Mitigating such attacks in real time requiresselection and coordinated deployment of counter measures across thenetwork in a manner that maximizes protection of network assets whilecausing minimal disruption in network services.

In one example situation, at the inception of an attack a valuable butvulnerable asset can be isolated from the network, in which case theasset is preserved but its function becomes unavailable for the durationof the isolation. Efficient selection and deployment of countermeasuresis predicated on timely and reliable malware evaluation which includesa) using the already detected and/or suspected malware elements topredict the intended assemblies and b) assessing the degree of threatposed by the anticipated assemblies to various network assets. Whenemploying the self-adaptive multi modal data stream processing system102, the prediction process will be accounting for the composition ofmalware elements as well as the geospatial distribution of the sources,timing and the degree of correlation between the attack episodes. In theprocess, combinations of spatial, temporal and feature neurons(responding to different malware elements) represent different attackpatterns and the ways those patterns can unfold in time and across thenetwork. Attack predictions and response planning based on such patternscan be more expedient and reliable than when derived solely from malwarecomposition.

A variety of techniques for attack detection and response optimizationare known in the art (for example, Topological Vulnerability Analysis,etc.). In one embodiment, a process of the self-adaptive multi modaldata stream processing system 102 (understanding-based cyber security,or UBCS) can be implemented independently or in conjunction with suchanalysis techniques, with the possibility of yielding radicalimprovements in the analysis outcome. According to one embodiment,implementing UBCS starts with defining and associating feature neuronsand spatial neurons with sensors and geospatial locations in the cybernetwork, correspondingly. When applying UBCS, the neurons will be linkedinto a virtual network, with weights on the links determined by therelative frequency of neuron co-activation and/or other factors. WhenUBCS is used in conjunction with other techniques, the links and linkweights in the virtual network can be derived from simulations andanalysis involving sample attack scenarios. Alternatively, they can bederived from the actual history of the cyber network and/or supplied byanalysts. Also, all three methods can be combined so that the initialvalues supplied by analysts get adjusted based on the simulation andanalysis as well as the actual history.

Partitioning the virtual network into packets reveals correlationbetween malware elements, geospatial locations and temporal organizationof attack patterns across a range of scenarios. Obtaining suchcorrelations reduces evaluation uncertainty (e.g., a suspicious stringis likely to be a malware element if arriving from servers in the area 1and less likely so if arriving from server in the area 2) and improvesprediction reliability (detecting some constituents of a packet predictsarrival of the other constituents) and expected response efficiency(e.g., reliable prediction enables preemptive countermeasures). As theinformation accumulates, the network re-partitions producing new packetsand adjusting the previously formed ones. Energy barriers in the networkprovide a mechanism for optimizing network stability, that is,preventing both responding to insignificant changes in neuronalactivation and ignoring the significant ones.

Complex and hyper complex neurons in the virtual hierarchy furtherimprove evaluation of and response construction for attack patternshaving complex temporal, spatial and feature organization. Semanticneurons can capture global relations in the cyber network and displaythem in a manner facilitating situational understanding by humananalysts (e.g., attack is being mounted from servers in the Region A,with the assets x, y, z being the likely targets).

FIG. 3 is a flow diagram illustrating a process 300 of the multi modaldata stream processing system 102, according to one illustratedembodiment.

At 302, the construction module 108 under control of control module 104of the multi modal data stream processing system 102 constructs a modelof a situation built upon an underlying associative neural networkstored in the associative memory module 106 to establish situationalunderstanding of the situation, wherein the model is comprised ofinvariant and variable entities and relationships between the entities,wherein each entity is able to be nested by the control module by beingcomprised of lower level models and wherein the lower level models areformed of packets and are groups of packets.

At 304, the control module 104 of the multi modal data stream processingsystem 102 manipulates the lower level models, leaving the underlyingassociative neural network intact by not changing synaptic weights inthe underlying associative neural network in the manipulation of thelower level models.

At 306, the multi modal data stream processing system 102 reduces energyconsumption and energy dissipation in the constructing and themanipulating of the models by the control module 104 seekingprogressively more general and adequate models persisting throughvarious situations and wherein the reducing energy consumption anddissipation translates into negentropy production.

FIG. 4 is a flow diagram illustrating a process 400 of the multi modaldata stream processing system 102, including details regarding howassociative artificial neural networks of the system are developed,according to one illustrated embodiment.

At 402, the construction module 108, under control of control module 104of the multi modal data stream processing system 102, correspondsindividual nodes of link-weighted associative artificial neural networksto respective artificial neurons of a plurality of artificial neuronsthat respond to different data elements of data representing a pluralityof different data streams representing a situation.

At 404, the construction module 108 under control of control module 104of the multi modal data stream processing system 102 establishes linkweights of the link-weighted associative artificial neural networkswhich represent a frequency of co-occurrence of the different dataelements of the data representing the plurality of different datastreams.

At 406, the control module 104 of the multi modal data stream processingsystem 102 dynamically partitions, as a situation unfolds over time, thelink-weighted associative artificial neural networks into internallycohesive subnetworks and externally weakly coupled subnetworks, whereinthe subnetworks are packets, each corresponding to at least a respectiveone of various different combinations of the data elements.

At 408, the control module 104 of the multi modal data stream processingsystem 102 performs dynamic mapping between the packets as the situationunfolds over time to adjust at least one three-partite situation modelto improve the at least one three-partite situation model for use inunderstanding of the situation, wherein the partitioning and dynamicmapping leave the artificial neural network intact by not changingsynaptic weights in the artificial neural network in the partitioningand the dynamic mapping.

At 410, the multi modal data stream processing system 102, based on theat least one three-partite situation model, generates situationalunderstanding of the situation.

At 412, the multi modal data stream processing system 102, reducesenergy consumption and dissipation in the constructing, partitioning anddynamically mapping by the control module 104 seeking progressively moregeneral and adequate models persisting through various situations andwherein the reducing energy consumption and dissipation translates intonegentropy production.

FIG. 5 is a flow diagram illustrating a process 500 of the multi modaldata stream processing system 102 including details regardingdynamically partitioning link-weighted associative artificial neuralnetworks of the system, according to one illustrated embodiment.

At 502, the control of control module 104 of the multi modal data streamprocessing system 102 dynamically partitions, as a situation representedby a plurality of different data streams unfolds over time,link-weighted associative artificial neural networks into internallycohesive subnetworks and externally weakly coupled subnetworks, whereinthe subnetworks are packets, each corresponding to at least a respectiveone of various different combinations of data elements of datarepresenting the plurality of different data streams.

At 504, the control module 104 of the multi modal data stream processingsystem 102 performs dynamic mapping between the packets as the situationunfolds over time to adjust at least one three-partite situation modelto improve the at least one three-partite situation model for use inunderstanding of the situation by the system, wherein the partitioningand dynamic mapping leave the artificial neural network intact by notchanging synaptic weights in the artificial neural network in thepartitioning and the dynamic mapping.

At 506, the multi modal data stream processing system 102 generatessituational understanding of the situation based on the at least onethree-partite situation model.

At 508, the multi modal data stream processing system 102 reduces energyconsumption and dissipation in the partitioning and the dynamicallymapping by seeking progressively more general and adequate modelspersisting through various situations wherein the reducing energyconsumption and dissipation translates into negentropy production.

FIG. 6 is a flow diagram illustrating a process 600 of the multi modaldata stream processing system 102 including details regarding improvinga three-partite situation model of the system for use in understandingof a situation represented by the model, according to one illustratedembodiment.

At 602, the construction module 108, under control of the control module104 of the multi modal data stream processing system 102, constructs atleast one three-partite situation model of a situation represented by aplurality of different data streams by making associations of artificialneurons of a plurality of artificial neurons of various different typesin an artificial neural network in a spatiotemporal associative memory.

At 604, the control module 104 of the multi modal data stream processingsystem 102 dynamically partitions, as the situation unfolds over time,link-weighted associative artificial neural networks of the artificialneural network into packets, each corresponding to at least a respectiveone of various different combinations of data elements of datarepresenting the plurality of different data streams.

At 606, the control module 104 of the multi modal data stream processingsystem 102 performs dynamic mapping between the packets as the situationunfolds over time to adjust the at least one three-partite situationmodel to improve the at least one three-partite situation model for usein understanding of the situation, wherein the constructing, dynamicallypartitioning and dynamic mapping leave the artificial neural networkintact by not changing synaptic weights in the artificial neural networkin the constructing, partitioning and the dynamic mapping.

At 608, the multi modal data stream processing system 102 generatessituational understanding of the situation as the situation unfolds overtime based on the at least one three-partite situation model.

At 610, the multi modal data stream processing system 102 reduces energyconsumption and dissipation in the constructing, dynamicallypartitioning and dynamically mapping by seeking progressively moregeneral and adequate models persisting through various situationswherein the reducing energy consumption and dissipation translates intonegentropy production.

Human understanding results from creating and manipulating mentalmodels. In one embodiment, the self-adaptive multi modal data streamprocessing system 102, creation and manipulation involve “neuronalpackets” formed in mutually constraining associative networks of fourbasic types as described above. The process of the self-adaptive multimodal data stream processing system 102 is thermodynamically driven,striving to produce a minimal number of maximally stable models.Obtaining such models is experienced by the system 102 as grasping, orunderstanding the input stream (objects, their relations and the flow ofchanges).

The foregoing detailed description has set forth various embodiments ofthe devices and/or processes via the use of block diagrams, schematics,and examples. Insofar as such block diagrams, schematics, and examplescontain one or more functions and/or operations, it will be understoodby those skilled in the art that each function and/or operation withinsuch block diagrams, flowcharts, or examples can be implemented,individually and/or collectively, by a wide range of hardware, software,firmware, or virtually any combination thereof. In one embodiment, thepresent subject matter may be implemented via Application SpecificIntegrated Circuits (ASICs). However, those skilled in the art willrecognize that the embodiments disclosed herein, in whole or in part,can be equivalently implemented in standard integrated circuits, as oneor more computer programs running on one or more computers (e.g., as oneor more programs running on one or more computer systems), as one ormore programs running on one or more controllers (e.g.,microcontrollers) as one or more programs running on one or moreprocessors (e.g., microprocessors), as firmware, or as virtually anycombination thereof, and that designing the circuitry and/or writing thecode for the software and or firmware would be well within the skill ofone of ordinary skill in the art in light of this disclosure.

In addition, those skilled in the art will appreciate that themechanisms taught herein are capable of being distributed as a programproduct in a variety of forms, and that an illustrative embodimentapplies equally regardless of the particular type of signal bearingmedia used to actually carry out the distribution. Examples of signalbearing media include, but are not limited to, the following: recordabletype media such as hard disk drives, DVD ROMs, solid state drives, flashmemory drives, associative memory and other computer memory.

The various embodiments described above can be combined to providefurther embodiments. Aspects of the embodiments can be modified, ifnecessary, to employ systems, circuits and concepts to provide yetfurther embodiments.

These and other changes can be made to the embodiments in light of theabove-detailed description. In general, in the following claims, theterms used should not be construed to limit the claims to the specificembodiments disclosed in the specification and the claims, but should beconstrued to include all possible embodiments along with the full scopeof equivalents to which such claims are entitled. Accordingly, theclaims are not limited by the disclosure.

1-25. (canceled)
 26. A method of obtaining understanding of multimodaldata streams by dynamically optimizing allocation of neuronal resourcesto data elements in the stream wherein resources are drawn from neuronalpool on which four temporal constraints are defined: pool's longevitylimit (life span), neuronal recuperation period, mobilization duration,and link decay period.
 27. The method of claim 25 wherein the survivalminimum energy inflow is defined on the pool and resource optimizationseeks to maintain energy inflow at or above the survival minimum for theduration of the life span.
 28. The method of claim 25 wherein dynamicresource optimization seeks to maximize internally generated negentropyin the neuronal pool computed as a function of the relative increase ofthe connected structure in the pool and decrease in the frequency ofstructure modifications in the course of allocations. (or as a functionof the number and stability (degree of invariance) of the unifyingmodels constructed in the pool).
 29. The method of claim 25 whereinoptimization involves packet adjustments including a) selectivelydissociating some neuronal packets from the rest of the virtual network,b) unfolding dissociated packets and re-distributing neurons, and c)enfolding the resulting packets.
 30. The method of claim 25 whereinmodels are adjusted by inserting/removing links and/or nodes in thenetworks, adjusting link weights, expanding/shrinking packets, other inorder to improve the model's predictive and retrodictive performance andthus achieve a higher degree of situation understanding.
 31. The methodof claim 25 wherein adjustment operations by the Control Module engagecontrol neurons and involve inhibiting/activating individual neurons,changing response characteristics (probabilities) of individual neurons,re-distributing neurons between packets, re-distributing packets betweenentities.
 32. The method of claim 25 wherein formation of neuronalpackets and coordination of packet firing (coordinated rotation ofpacket vectors) in the neuronal pool are subject to temporal constraintsincluding: longevity limit θ₁ determining the time period during whichneurons remain capable of firing (all firing ceases after θ₁);recuperation period θ₂ (having fired, neuron can resume firing no soonerthan after θ₂), mobilization (recruitment) rate θ₃ (the average numberof neurons or neuronal packets across the pool that can commence firingin a unit time); connectivity decay period θ₄ (inter-neuron connectionstrength (weight) decays to zero in the absence of co-firing during θ₄).33. The method of claim 25 wherein the system for understandingmultimodal data streams is applied for situational control of autonomousvehicles comprising a) processing sensor information from sensorsmounted on the vehicle, b) applying information to form situation modelsdefining objects in the vehicle surrounds and relations between the saidobjects comprising behavior of the objects and the mutual behaviorconstraints, c) applying the model to predict likely events resultingfrom behavior changes and d) computing control signals for theautonomous vehicle based on the said predictions.
 34. Neuronal pool,comprising groups of identical type neurons (minicolumns) such that a)neuronal packets are formed of neurons mobilized from differentminicolumns; b) functional distance in the neuronal space betweenneuronal packets A and B is determined by the degree of overlapaccounting for the number of neuron types present in A but not in B, thenumber of neuron type present in B but not in A, and the number ofneuron types present in both A and B; and c) the degree of similaritybetween objects represented by neuronal packets A and B is determined byfunctional distance between A and B.